Skip to content

Support Zookeeperless NiFi #775

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 26 commits into from
May 27, 2025
Merged

Support Zookeeperless NiFi #775

merged 26 commits into from
May 27, 2025

Conversation

nightkr
Copy link
Member

@nightkr nightkr commented Apr 9, 2025 •
edited by maltesander
Loading

Description

Fixes #737.

This makes spec.clusterConfig.zookeeperConfigMapName optional, switching to the Kubernetes provider if it is not set. The prose docs mention both options, but the quick start guide ("First steps") sticks to ZooKeeper, since it still also recommends NiFi 1.x.

A potential v1alpha2 could change this to something like

spec:
  clusterConfig:
    clusterBackend:
      zookeeper:
        configMapName: ...
      # or...
      kubernetes: {}

But I don't think it's worth the compatibility break for v1alpha1.

Definition of Done Checklist

  • Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
  • Please make sure all these things are done and tick the boxes
# Author
- [X] Changes are OpenShift compatible
- [X] CRD changes approved
- [X] CRD documentation for all fields, following the [style guide](https://docs.stackable.tech/home/nightly/contributor/docs/style-guide).
- [X] Helm chart can be installed and deployed operator works
- [X] Integration tests passed (for non trivial changes)
- [X] Changes need to be "offline" compatible

# Reviewer
- [x] Code contains useful comments
- [x] Code contains useful logging statements
- [x] (Integration-)Test cases added
- [x] Documentation added or updated. Follows the [style guide](https://docs.stackable.tech/home/nightly/contributor/docs/style-guide).
- [x] Changelog updated
- [x] Cargo.toml only contains references to git tags (not specific commits or branches)

# Acceptance
- [ ] Feature Tracker has been updated
- [ ] Proper release label has been added
- [ ] [Roadmap](https://github.com/orgs/stackabletech/projects/25/views/1) has been updated

@nightkr nightkr self-assigned this Apr 9, 2025
@nightkr
Copy link
Member Author

nightkr commented Apr 9, 2025

Moving the CRD changes to RfC.

nightkr added 2 commits April 9, 2025 12:02
@nightkr
Rename clustering_mode to clustering_backend
1aa50cf
@nightkr
Make the nifi prepare script more idempotent
f1a3368 
Previously, these bits would error when re-executed, causing confusing
errors when templating fails.
@nightkr
Copy link
Member Author

nightkr commented May 6, 2025

Moving the CRD changes to voting.

@NickLarsenNZ
Copy link
Member

NickLarsenNZ commented May 6, 2025 •
edited
Loading

Sorry, didn't see that it was already RfC and now in voting

Sounds good to me. Moving to option is non-breaking.

What do we do in cases where it is None, but the NiFi version still requires it?
I assume we error out on reconcile.

@nightkr
Copy link
Member Author

nightkr commented May 6, 2025

In that case NiFi would crash on startup. We can add an explicit check for it.. maybe a bit overkill but if we're keeping 1.x for the relatively long haul then it might make sense?

@sbernauer
Copy link
Member

I think having a snafu Error like "zookeeper-less election not supported for version {}. It's only supported for NiFi 2.x" would be great!

nightkr added 3 commits May 6, 2025 13:28
@nightkr
Only include the used cluster state provider in xml config
db206b8
@nightkr
Add test
ef63fa9 
Copied from smoke since I'm not aware of a native way to exclude
specific dimension combinations.
@nightkr
Explicitly fail when trying to use NiFi 1.x in zookeeperless mode
4f63520
@nightkr
Copy link
Member Author

nightkr commented May 6, 2025

Added an error in 4f63520.

@nightkr nightkr marked this pull request as ready for review May 8, 2025 14:33
@nightkr
Copy link
Member Author

nightkr commented May 8, 2025

Tests: https://testing.stackable.tech/view/02%20Operator%20Tests%20(custom)/job/nifi-operator-it-custom/38/, https://testing.stackable.tech/view/02%20Operator%20Tests%20(custom)/job/nifi-operator-it-custom/39/

@nightkr
Copy link
Member Author

nightkr commented May 9, 2025

Moving the CRD change to approved.

All tests passed in Jenkins except for ldap_nifi-2.2.0_zookeeper-latest-3.9.3_ldap-use-tls-true_openshift-false, but that passed locally.

@nightkr nightkr requested a review from a team May 9, 2025 12:03
@soenkeliebau
Copy link
Member

I have looked over the code changes and it all looks good to me.
Haven't actually tested it though I have to admit..

maltesander
maltesander requested changes May 26, 2025
View reviewed changes
Copy link
Member

@maltesander maltesander left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Im not quite happy with duplicating the test like that but do not see any other way currently of avoiding useless test cases.
Do we want a nifi v1 deprecation ticket where we write down what needs to be done? (e.g. merge test cases from this PR).

maltesander
maltesander previously approved these changes May 27, 2025
View reviewed changes
Copy link
Member

@maltesander maltesander left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@nightkr nightkr enabled auto-merge May 27, 2025 14:59
@nightkr nightkr requested a review from maltesander May 27, 2025 14:59
maltesander
maltesander approved these changes May 27, 2025
View reviewed changes
Copy link
Member

@maltesander maltesander left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@nightkr nightkr added this pull request to the merge queue May 27, 2025
Merged via the queue into main with commit b985f03 May 27, 2025
16 of 17 checks passed
@nightkr nightkr deleted the feature/zkless branch May 27, 2025 15:18
@sbernauer sbernauer mentioned this pull request May 28, 2025
Closed
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@maltesander maltesander maltesander approved these changes

@soenkeliebau soenkeliebau Awaiting requested review from soenkeliebau

Assignees

@nightkr nightkr

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Support for running with Kubernetes as quorum manager instead of ZooKeeper
5 participants
@nightkr @NickLarsenNZ @sbernauer @soenkeliebau @maltesander